{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a65a02d9-52e0-4468-8242-979ccffa22ca",
   "metadata": {},
   "source": [
    "# Using different pre-defined difficulty as complexity measure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c5637077-9ede-45da-858d-6fa97f1337b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import re\n",
    "import openai\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from tqdm import tqdm\n",
    "from tenacity import retry, stop_after_attempt, wait_chain, wait_fixed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "58884122-7f39-42a8-badd-d1ec0d7fa907",
   "metadata": {},
   "outputs": [],
   "source": [
    "@retry(wait=wait_chain(*[wait_fixed(3) for i in range(3)] +\n",
    "                       [wait_fixed(5) for i in range(2)] +\n",
    "                       [wait_fixed(10)]))\n",
    "def completion_with_backoff(**kwargs):\n",
    "    return openai.Completion.create(**kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "49e02868-0f2e-41d6-918b-bbc538a9274d",
   "metadata": {},
   "outputs": [],
   "source": [
    "openai.api_key = \"sk-\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4e8f185e-d4bc-454e-8d17-e2073e646bc5",
   "metadata": {},
   "outputs": [],
   "source": [
    "math_algebra = []\n",
    "dir_name = 'MATH/train/algebra'\n",
    "for filename in os.listdir(dir_name):\n",
    "    if(filename.endswith('.json')):\n",
    "        d = json.load(open(dir_name + '/' + filename))\n",
    "        math_algebra.append(d)\n",
    "        \n",
    "dev_idx = np.load('MATH/train/algebra_dev_idx.npy')\n",
    "math_algebra_dev = [math_algebra[i] for i in dev_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "252308a9-363a-4191-bea0-95a0c4b4e682",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'problem': 'George has a quadratic of the form $x^2+bx+\\\\frac13$, where $b$ is a specific negative number. Using his knowledge of how to complete the square, George is able to rewrite this quadratic in the form $(x+m)^2+\\\\frac{1}{12}$. What is $b$?',\n",
       " 'level': 'Level 5',\n",
       " 'type': 'Algebra',\n",
       " 'solution': 'The expansion of $(x+m)^2+\\\\frac{1}{12}$ is $x^2+2mx+m^2+\\\\frac{1}{12}$, which has a constant term of $m^2+\\\\frac{1}{12}$. This constant term must be equal to the constant term of the original quadratic, so $$m^2+\\\\frac{1}{12} = \\\\frac13,$$and $$m^2 = \\\\frac13-\\\\frac{1}{12} = \\\\frac14.$$This yields the possibilities $m=\\\\frac12$ and $m=-\\\\frac12$.\\n\\nIf $m=\\\\frac12$, then $(x+m)^2+\\\\frac{1}{12} = x^2+x+\\\\frac14+\\\\frac{1}{12} = x^2+x+\\\\frac13$. This implies $b=1$, but we reject this possibility because we were told that $b$ is a negative number.\\n\\nIf $m=-\\\\frac12$, then $(x+m)^2+\\\\frac{1}{12} = x^2-x+\\\\frac14+\\\\frac{1}{12} = x^2-x+\\\\frac13$, giving the result $b=\\\\boxed{-1}$.'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "math_algebra_dev[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b0b77b5e-a48f-4710-8a97-0c3676b81d73",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_grade_1 = open('lib_prompt/algebra/prompt_grad_1.txt').read()\n",
    "prompt_grade_3 = open('lib_prompt/algebra/prompt_grad_3.txt').read()\n",
    "prompt_grade_5 = open('lib_prompt/algebra/prompt_grad_5.txt').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9826d8a3-ae9b-4d0e-8246-80eff0654c7c",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Kevin Kangaroo begins hopping on a number line at 0. He wants to get to 1, but he can hop only $\\frac{1}{3}$ of the distance. Each hop tires him out so that he continues to hop $\\frac{1}{3}$ of the remaining distance. How far has he hopped after five hops? Express your answer as a common fraction.\n",
      "Let's think step by step\n",
      "Kevin hops $1/3$ of the remaining distance with every hop. \n",
      "His first hop takes $1/3$ closer. \n",
      "For his second hop, he has $2/3$ left to travel, so he hops forward $(2/3)(1/3)$. \n",
      "For his third hop, he has $(2/3)^2$ left to travel, so he hops forward $(2/3)^2(1/3)$. \n",
      "In general, Kevin hops forward $(2/3)^{k-1}(1/3)$ on his $k$th hop. \n",
      "We want to find how far he has hopped after five hops. \n",
      "This is a finite geometric series with first term $1/3$, common ratio $2/3$, and five terms. \n",
      "Thus, Kevin has hopped $\\frac{\\frac{1}{3}\\left(1-\\left(\\frac{2}{3}\\right)^5\\right)}{1-\\frac{2}{3}} = \\boxed{\\frac{211}{243}}$.\n",
      "The answer is \\frac{211}{243}}\n",
      "\n",
      "Question: What is the area of the region defined by the equation $x^2+y^2 - 7 = 4y-14x+3$?\n",
      "Let's think step by step\n",
      "We rewrite the equation as $x^2 + 14x + y^2 - 4y = 10$ and then complete the square, \n",
      "resulting in  $(x+7)^2-49 + (y-2)^2-4=10$, \n",
      "or $(x+7)^2+(y-2)^2=63$. \n",
      "This is the equation of a circle with center $(-7, 2)$ and radius $\\sqrt{63},$ \n",
      "so the area of this region is $\\pi r^2 = \\boxed{63\\pi}$.\n",
      "The answer is 63\\pi\n",
      "\n",
      "Question: If $x^2+y^2=1$, what is the largest possible value of $|x|+|y|$?\n",
      "Let's think step by step\n",
      "If $(x,y)$ lies on the circle, \n",
      "so does $(x,-y),$ $(-x,-y),$ and $(-x,-y),$ (which all give the same value of $|x| + |y|$), \n",
      "so we can assume that $x \\ge 0$ and $y \\ge 0.$\n",
      "Then $|x| + |y| = x + y.$  Squaring, we get\n",
      "\\[(x + y)^2 = x^2 + 2xy + y^2 = 1 + 2xy.\\]\n",
      "Note that $(x - y)^2 \\ge 0.$  \n",
      "Expanding, we get $x^2 - 2xy + y^2 \\ge 0,$ so $2xy \\le x^2 + y^2 = 1.$  \n",
      "Hence,\\[1 + 2xy \\le 2,\\]which means $x + y \\le \\sqrt{2}.$  \n",
      "Equality occurs when $x = y = \\frac{1}{\\sqrt{2}},$ \n",
      "so the maximum value of $|x| + |y|$ is $\\boxed{\\sqrt{2}}.$\n",
      "The answer is \\sqrt{2}\n",
      "\n",
      "Question: If $f(x)=\\frac{ax+b}{cx+d}, abcd\\not=0$ and $f(f(x))=x$ for all $x$ in the domain of $f$, what is the value of $a+d$?\n",
      "Let's think step by step\n",
      "The condition $f(f(x))$ means that $f$ is the inverse of itself, \n",
      "so its graph is symmetrical about the line $y = x$. \n",
      "With a rational function of this form, we will have two asymptotes: \n",
      "a vertical one at $x=-d/c$ if $cx+d$ does not divide $ax+b$, \n",
      "and a horizontal one at $y=a/c$, \n",
      "if we take the limit of $f(x)$ as $x$ goes to $\\pm\\infty$. \n",
      "In order for $f$ to be its own inverse, the intersection of the asymptotes must lie on the line $y=x$ \n",
      "so that it and its asymptotes reflect onto themselves. \n",
      "This means that $-d/c=a/c$, \n",
      "and therefore $-d=a$ and $a+d=\\boxed{0}$.\n",
      "The answer is 0\n",
      "\n",
      "Question: A math teacher requires Noelle to do one homework assignment for each of the first five homework points she wants to earn; for each of the next five homework points, she needs to do two homework assignments; and so on, so that to earn the $n^{\\text{th}}$ homework point, she has to do $n\\div5$ (rounded up) homework assignments. For example, when she has 11 points, it will take $12\\div5=2.4\\rightarrow3$ homework assignments to earn her $12^{\\text{th}}$ point. What is the smallest number of homework assignments necessary to earn a total of 25 homework points?\n",
      "Let's think step by step\n",
      "Noelle only has to do 1 homework assignment to earn her first point, \n",
      "and the same is true for each of her first five points.  \n",
      "She must then do 2 homework assignments to earn her sixth point, seventh point, and so on, up to her tenth point. \n",
      "Continuing, we see that Noelle must do a total of \\[1+1+1+1+1+2+2+2+2+2+\\dots+5+5+5+5+5\\] homework assignments to earn 25 points.\n",
      "This sum may be rewritten as $5(1+2+3+4+5)=5(15)=\\boxed{75}$.\n",
      "The answer is 75\n",
      "\n",
      "Question: The quadratic equation $x^2+mx+n=0$ has roots that are twice those of $x^2+px+m=0,$ and none of $m,$ $n,$ and $p$ is zero. What is the value of $n/p?$\n",
      "Let's think step by step\n",
      "Let $r_1$ and $r_2$ be the roots of $x^2+px+m=0.$ \n",
      "Since the roots of $x^2+mx+n=0$ are $2r_1$ and $2r_2,$ we have the following relationships: \\[\n",
      "m=r_1 r_2,\\quad n=4r_1 r_2,\\quad p=-(r_1+r_2), \\quad\\text{and}\\quad\n",
      "m=-2(r_1+r_2).\n",
      "\\] So \\[\n",
      "n = 4m, \\quad p = \\frac{1}{2}m,\n",
      "\\quad\\text{and}\\quad\n",
      "\\frac{n}{p}=\\frac{4m}{\\frac{1}{2}m}=\\boxed{8}.\n",
      "\\]\n",
      "Alternatively, the roots of \\[\n",
      "\\left(\\frac{x}{2}\\right)^2 + p\\left(\\frac{x}{2}\\right) + m = 0\n",
      "\\] are twice those of $x^2 + px + m = 0.$ \n",
      "Since the first equation is equivalent to $x^2 + 2px + 4m = 0,$ \n",
      "we have \\[m = 2p \\quad\\text{and}\\quad n = 4m, \\quad\\text{so}\\quad \\frac{n}{p} = \\boxed{8}.\\]\n",
      "The answer is 8\n",
      "\n",
      "Question: Expand $(2z^2 + 5z - 6)(3z^3 - 2z + 1)$.\n",
      "Let's think step by step\n",
      "$$\\begin{array}{crrrrrrr}\n",
      "& & & 3z^3 & & -2z & + 1 & \\\\\n",
      "\\times & & & & 2z^2 & +5z & -6 \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& & & -18z^3 & & +12z & -6 & \\\\\n",
      "& & +15z^4 & & -10z^2 & +5z & & \\\\\n",
      "+ & 6z^5 & & -4z^3 & +2z^2 & & & \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& 6z^5 & +15z^4 & -22z^3 & - 8z^2 &+17z & -6 &\n",
      "\\end{array}$$ \n",
      "The answer is 6z^5+15z^4-22z^3-8z^2+17z-6}.\n",
      "\n",
      "Question: Find the mean of all solutions for $x$ when $x^3 + 3x^2 - 10x = 0$.\n",
      "Let's think step by step\n",
      "First, we factor the equation as $x(x^2 +3x - 10) = 0$.  \n",
      "So, one solution is $x=0$ and the other two solutions are the solutions to $x^2 + 3x-10=0$.\n",
      "We could either factor the quadratic, or note that the sum of the solutions to this quadratic is $-(3/1)=-3$, \n",
      "so the mean of the three solutions to the original equation is $-3/3=\\boxed{-1}$.\n",
      "The answer is -1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_grade_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "16a0027e-6086-417a-9569-95234f3e170b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_answer(s):\n",
    "    assert('boxed' in s)\n",
    "    ans = s.split('boxed')[-1]\n",
    "    if(ans[0] == '{'):\n",
    "        stack = 1\n",
    "        a = ''\n",
    "        for c in ans[1:]:\n",
    "            if(c == '{'): \n",
    "                stack += 1\n",
    "                a += c\n",
    "            elif(c == '}'): \n",
    "                stack -= 1\n",
    "                if(stack == 0): break\n",
    "                a += c\n",
    "            else: \n",
    "                a += c\n",
    "    else:\n",
    "        a = ans.split('$')[0].strip()\n",
    "    return a\n",
    "\n",
    "def test_answer(pred_str, ans_str):\n",
    "    if('The answer is ' in pred_str):\n",
    "        pred = pred_str.split('The answer is ')[-1].strip()\n",
    "    else:\n",
    "        pattern = '\\d*\\.?\\d+'\n",
    "        pred = re.findall(pattern, pred_str)\n",
    "        if(len(pred) >= 1):\n",
    "            # print(pred_str)\n",
    "            pred = pred[-1]\n",
    "        else: pred = ''\n",
    "\n",
    "    gold = find_answer(ans_str)\n",
    "    # gold = re.findall(pattern, ans_str)\n",
    "    # print(ans_str)\n",
    "    # gold = gold[-1]\n",
    "    # print('pred:', pred)\n",
    "    # print('gold:', gold)\n",
    "    # print('---\\n\\n')\n",
    "    return pred == gold\n",
    "\n",
    "def parse_pred_ans(filename):\n",
    "    with open(filename) as fd: lines = fd.readlines()\n",
    "    am, a = None, None\n",
    "    num_q, acc = 0, 0\n",
    "    current_mode = 'none'\n",
    "    questions = []\n",
    "    ans_pred = []\n",
    "    ans_gold = []\n",
    "    for l in lines:\n",
    "        if(l.startswith('Q: ')):\n",
    "            if(am is not None and a is not None):\n",
    "                questions.append(q)\n",
    "                ans_pred.append(am)\n",
    "                ans_gold.append(a)\n",
    "                if(test_answer(am, a)):\n",
    "                    acc += 1\n",
    "            current_mode = 'q'\n",
    "            q = l\n",
    "            num_q += 1\n",
    "        elif(l.startswith('A_model:')):\n",
    "            current_mode = 'am'\n",
    "            am = l\n",
    "        elif(l.startswith('A:')):\n",
    "            current_mode = 'a'\n",
    "            a = l\n",
    "        else:\n",
    "            if(current_mode == 'q'): q += l\n",
    "            elif(current_mode == 'am'): am += l\n",
    "            elif(current_mode == 'a'): a += l\n",
    "            else:\n",
    "                raise ValueError(current_mode)\n",
    "                \n",
    "    questions.append(q)\n",
    "    ans_pred.append(am)\n",
    "    ans_gold.append(a)\n",
    "    if(test_answer(am, a)):\n",
    "        acc += 1\n",
    "    print('num_q %d correct %d ratio %.4f' % (num_q, acc, float(acc / num_q)))\n",
    "    return questions, ans_pred, ans_gold"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b95a32bb-e6e3-4f50-a292-6b270c52f4d3",
   "metadata": {},
   "source": [
    "# Grade 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eadc62c-4055-4d02-90f5-2a227f4e14a9",
   "metadata": {},
   "source": [
    "## Text-davinci-002, Acc 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "66dd127d-6938-4d12-9118-792b108abcce",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [18:36<00:00,  5.58s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_1.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_1 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = openai.Completion.create(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "e43cee89-c6a7-4081-a2c8-f1e20f08429b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 36 ratio 0.1800\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_1.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297fcd1a-d488-499f-921f-ed66e92f5a21",
   "metadata": {},
   "source": [
    "## Code-davinci-002, Acc 24"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "199ebdc9-5d23-4430-8a6c-200c7ce5b3a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_ans(ans_model):\n",
    "    ans_model = ans_model.split('\\n')\n",
    "    ans = []\n",
    "    residual = []\n",
    "    for li, al in enumerate(ans_model):\n",
    "        ans.append(al)\n",
    "        if('answer is' in al):\n",
    "            break\n",
    "    residual = list(ans_model[li + 1:])\n",
    "    ans = '\\n'.join(ans)\n",
    "    residual = '\\n'.join(residual)\n",
    "    return ans, residual"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "97e23e01-a5da-4500-98a3-3f2ae7dee220",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [35:08<00:00, 10.54s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_1_code_davinci_002.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_1 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = completion_with_backoff(model=\"code-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        ans_, residual = extract_ans(ans_model)\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "53daa3b2-c143-45d3-a2fd-adb1eccabac2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 48 ratio 0.2400\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_1_code_davinci_002.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a304959c-944e-4463-afa4-b131887e141d",
   "metadata": {},
   "source": [
    "## Text-davinci-003, Acc 22.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d7859d4e-3973-436a-8276-871ac31c8ae9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [23:40<00:00,  7.10s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_1_text_davinci_003.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_1 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = openai.Completion.create(model=\"text-davinci-003\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d118bf19-3cdf-4e2d-852f-b139663741f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 45 ratio 0.2250\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_1_text_davinci_003.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e4dc90f-ad52-41a1-a108-df9420f8ec50",
   "metadata": {},
   "source": [
    "# Grade 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a220449d-a8b2-4ccc-90a0-fe3e99a04bc5",
   "metadata": {},
   "source": [
    "## Text-davinci-002, Acc 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "279ceb48-0677-4173-a7e6-759338dd7641",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [20:05<00:00,  6.03s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_3.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_3 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = openai.Completion.create(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "f2b05940-ab1f-485a-8ec9-850ddcb43812",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 36 ratio 0.1800\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_3.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24d18cfe-38ba-4bc2-8e91-673b7b2caceb",
   "metadata": {},
   "source": [
    "# Grade 5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82423616-1076-4cea-88d3-8cbfe1a22111",
   "metadata": {},
   "source": [
    "## Text-davinci-002, Acc 17"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "69b43793-b100-4056-84df-5b2004b8879b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [19:48<00:00,  5.94s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_5.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_5 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = openai.Completion.create(model=\"text-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1d8dd65f-4796-4490-ad80-3c28324e62b9",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Kevin Kangaroo begins hopping on a number line at 0. He wants to get to 1, but he can hop only $\\frac{1}{3}$ of the distance. Each hop tires him out so that he continues to hop $\\frac{1}{3}$ of the remaining distance. How far has he hopped after five hops? Express your answer as a common fraction.\n",
      "Let's think step by step\n",
      "Kevin hops $1/3$ of the remaining distance with every hop. \n",
      "His first hop takes $1/3$ closer. \n",
      "For his second hop, he has $2/3$ left to travel, so he hops forward $(2/3)(1/3)$. \n",
      "For his third hop, he has $(2/3)^2$ left to travel, so he hops forward $(2/3)^2(1/3)$. \n",
      "In general, Kevin hops forward $(2/3)^{k-1}(1/3)$ on his $k$th hop. \n",
      "We want to find how far he has hopped after five hops. \n",
      "This is a finite geometric series with first term $1/3$, common ratio $2/3$, and five terms. \n",
      "Thus, Kevin has hopped $\\frac{\\frac{1}{3}\\left(1-\\left(\\frac{2}{3}\\right)^5\\right)}{1-\\frac{2}{3}} = \\boxed{\\frac{211}{243}}$.\n",
      "The answer is \\frac{211}{243}}\n",
      "\n",
      "Question: What is the area of the region defined by the equation $x^2+y^2 - 7 = 4y-14x+3$?\n",
      "Let's think step by step\n",
      "We rewrite the equation as $x^2 + 14x + y^2 - 4y = 10$ and then complete the square, \n",
      "resulting in  $(x+7)^2-49 + (y-2)^2-4=10$, \n",
      "or $(x+7)^2+(y-2)^2=63$. \n",
      "This is the equation of a circle with center $(-7, 2)$ and radius $\\sqrt{63},$ \n",
      "so the area of this region is $\\pi r^2 = \\boxed{63\\pi}$.\n",
      "The answer is 63\\pi\n",
      "\n",
      "Question: If $x^2+y^2=1$, what is the largest possible value of $|x|+|y|$?\n",
      "Let's think step by step\n",
      "If $(x,y)$ lies on the circle, \n",
      "so does $(x,-y),$ $(-x,-y),$ and $(-x,-y),$ (which all give the same value of $|x| + |y|$), \n",
      "so we can assume that $x \\ge 0$ and $y \\ge 0.$\n",
      "Then $|x| + |y| = x + y.$  Squaring, we get\n",
      "\\[(x + y)^2 = x^2 + 2xy + y^2 = 1 + 2xy.\\]\n",
      "Note that $(x - y)^2 \\ge 0.$  \n",
      "Expanding, we get $x^2 - 2xy + y^2 \\ge 0,$ so $2xy \\le x^2 + y^2 = 1.$  \n",
      "Hence,\\[1 + 2xy \\le 2,\\]which means $x + y \\le \\sqrt{2}.$  \n",
      "Equality occurs when $x = y = \\frac{1}{\\sqrt{2}},$ \n",
      "so the maximum value of $|x| + |y|$ is $\\boxed{\\sqrt{2}}.$\n",
      "The answer is \\sqrt{2}\n",
      "\n",
      "Question: If $f(x)=\\frac{ax+b}{cx+d}, abcd\\not=0$ and $f(f(x))=x$ for all $x$ in the domain of $f$, what is the value of $a+d$?\n",
      "Let's think step by step\n",
      "The condition $f(f(x))$ means that $f$ is the inverse of itself, \n",
      "so its graph is symmetrical about the line $y = x$. \n",
      "With a rational function of this form, we will have two asymptotes: \n",
      "a vertical one at $x=-d/c$ if $cx+d$ does not divide $ax+b$, \n",
      "and a horizontal one at $y=a/c$, \n",
      "if we take the limit of $f(x)$ as $x$ goes to $\\pm\\infty$. \n",
      "In order for $f$ to be its own inverse, the intersection of the asymptotes must lie on the line $y=x$ \n",
      "so that it and its asymptotes reflect onto themselves. \n",
      "This means that $-d/c=a/c$, \n",
      "and therefore $-d=a$ and $a+d=\\boxed{0}$.\n",
      "The answer is 0\n",
      "\n",
      "Question: A math teacher requires Noelle to do one homework assignment for each of the first five homework points she wants to earn; for each of the next five homework points, she needs to do two homework assignments; and so on, so that to earn the $n^{\\text{th}}$ homework point, she has to do $n\\div5$ (rounded up) homework assignments. For example, when she has 11 points, it will take $12\\div5=2.4\\rightarrow3$ homework assignments to earn her $12^{\\text{th}}$ point. What is the smallest number of homework assignments necessary to earn a total of 25 homework points?\n",
      "Let's think step by step\n",
      "Noelle only has to do 1 homework assignment to earn her first point, \n",
      "and the same is true for each of her first five points.  \n",
      "She must then do 2 homework assignments to earn her sixth point, seventh point, and so on, up to her tenth point. \n",
      "Continuing, we see that Noelle must do a total of \\[1+1+1+1+1+2+2+2+2+2+\\dots+5+5+5+5+5\\] homework assignments to earn 25 points.\n",
      "This sum may be rewritten as $5(1+2+3+4+5)=5(15)=\\boxed{75}$.\n",
      "The answer is 75\n",
      "\n",
      "Question: The quadratic equation $x^2+mx+n=0$ has roots that are twice those of $x^2+px+m=0,$ and none of $m,$ $n,$ and $p$ is zero. What is the value of $n/p?$\n",
      "Let's think step by step\n",
      "Let $r_1$ and $r_2$ be the roots of $x^2+px+m=0.$ \n",
      "Since the roots of $x^2+mx+n=0$ are $2r_1$ and $2r_2,$ we have the following relationships: \\[\n",
      "m=r_1 r_2,\\quad n=4r_1 r_2,\\quad p=-(r_1+r_2), \\quad\\text{and}\\quad\n",
      "m=-2(r_1+r_2).\n",
      "\\] So \\[\n",
      "n = 4m, \\quad p = \\frac{1}{2}m,\n",
      "\\quad\\text{and}\\quad\n",
      "\\frac{n}{p}=\\frac{4m}{\\frac{1}{2}m}=\\boxed{8}.\n",
      "\\]\n",
      "Alternatively, the roots of \\[\n",
      "\\left(\\frac{x}{2}\\right)^2 + p\\left(\\frac{x}{2}\\right) + m = 0\n",
      "\\] are twice those of $x^2 + px + m = 0.$ \n",
      "Since the first equation is equivalent to $x^2 + 2px + 4m = 0,$ \n",
      "we have \\[m = 2p \\quad\\text{and}\\quad n = 4m, \\quad\\text{so}\\quad \\frac{n}{p} = \\boxed{8}.\\]\n",
      "The answer is 8\n",
      "\n",
      "Question: Expand $(2z^2 + 5z - 6)(3z^3 - 2z + 1)$.\n",
      "Let's think step by step\n",
      "$$\\begin{array}{crrrrrrr}\n",
      "& & & 3z^3 & & -2z & + 1 & \\\\\n",
      "\\times & & & & 2z^2 & +5z & -6 \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& & & -18z^3 & & +12z & -6 & \\\\\n",
      "& & +15z^4 & & -10z^2 & +5z & & \\\\\n",
      "+ & 6z^5 & & -4z^3 & +2z^2 & & & \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& 6z^5 & +15z^4 & -22z^3 & - 8z^2 &+17z & -6 &\n",
      "\\end{array}$$ \n",
      "The answer is 6z^5+15z^4-22z^3-8z^2+17z-6}.\n",
      "\n",
      "Question: Find the mean of all solutions for $x$ when $x^3 + 3x^2 - 10x = 0$.\n",
      "Let's think step by step\n",
      "First, we factor the equation as $x(x^2 +3x - 10) = 0$.  \n",
      "So, one solution is $x=0$ and the other two solutions are the solutions to $x^2 + 3x-10=0$.\n",
      "We could either factor the quadratic, or note that the sum of the solutions to this quadratic is $-(3/1)=-3$, \n",
      "so the mean of the three solutions to the original equation is $-3/3=\\boxed{-1}$.\n",
      "The answer is -1\n",
      "\n",
      "Question: If $f(x)=\\frac{x+4}{x^2+ax+b}$, and $f(x)$ has two vertical asymptotes at $x=1$ and $x=-2$, find the sum of $a$ and $b$.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "46921dba-935a-419e-bc6c-7bc300ebe1df",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 34 ratio 0.1700\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_5.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "05c96c37-2ef3-4ed1-b765-cfc40b460482",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A_model:\n",
      "Let's think step by step\n",
      "The midpoint of the line segment is located at $(1, -2)$. \n",
      "This means that the average of the $x$-coordinates is 1 and the average of the $y$-coordinates is -2. \n",
      "We know that one endpoint is $(4, 5)$, so the other endpoint must be $(-2, 3)$.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(ans_pred[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e820af17-479a-4cbf-b3b6-f1beb74d25da",
   "metadata": {},
   "outputs": [],
   "source": [
    "pattern = '\\d*\\.?\\d+'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "418e927a-d002-45b9-a61d-a19e31511437",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['1', '2', '1', '2', '4', '5', '2', '3']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "re.findall(pattern, ans_pred[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7e06894-86a6-428c-b8e8-56c130366ab4",
   "metadata": {},
   "source": [
    "## Code-davinci-002, Acc 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fcfc95a5-e782-4108-9e22-b5c7e1d1ec21",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [34:50<00:00, 10.45s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_5_code_davinci_002.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_5 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = completion_with_backoff(model=\"code-davinci-002\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        ans_, residual = extract_ans(ans_model)\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0a79d214-4f8a-48c1-9a22-7128c87a8969",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 40 ratio 0.2000\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_5_code_davinci_002.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf9b4102-cbab-48c1-819f-7f23b0840c41",
   "metadata": {},
   "source": [
    "## Text-davinci-003, Acc 20.50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "5289ff09-9418-4bb3-b4e1-751db93ee14f",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: Kevin Kangaroo begins hopping on a number line at 0. He wants to get to 1, but he can hop only $\\frac{1}{3}$ of the distance. Each hop tires him out so that he continues to hop $\\frac{1}{3}$ of the remaining distance. How far has he hopped after five hops? Express your answer as a common fraction.\n",
      "Let's think step by step\n",
      "Kevin hops $1/3$ of the remaining distance with every hop. \n",
      "His first hop takes $1/3$ closer. \n",
      "For his second hop, he has $2/3$ left to travel, so he hops forward $(2/3)(1/3)$. \n",
      "For his third hop, he has $(2/3)^2$ left to travel, so he hops forward $(2/3)^2(1/3)$. \n",
      "In general, Kevin hops forward $(2/3)^{k-1}(1/3)$ on his $k$th hop. \n",
      "We want to find how far he has hopped after five hops. \n",
      "This is a finite geometric series with first term $1/3$, common ratio $2/3$, and five terms. \n",
      "Thus, Kevin has hopped $\\frac{\\frac{1}{3}\\left(1-\\left(\\frac{2}{3}\\right)^5\\right)}{1-\\frac{2}{3}} = \\boxed{\\frac{211}{243}}$.\n",
      "The answer is \\frac{211}{243}}\n",
      "\n",
      "Question: What is the area of the region defined by the equation $x^2+y^2 - 7 = 4y-14x+3$?\n",
      "Let's think step by step\n",
      "We rewrite the equation as $x^2 + 14x + y^2 - 4y = 10$ and then complete the square, \n",
      "resulting in  $(x+7)^2-49 + (y-2)^2-4=10$, \n",
      "or $(x+7)^2+(y-2)^2=63$. \n",
      "This is the equation of a circle with center $(-7, 2)$ and radius $\\sqrt{63},$ \n",
      "so the area of this region is $\\pi r^2 = \\boxed{63\\pi}$.\n",
      "The answer is 63\\pi\n",
      "\n",
      "Question: If $x^2+y^2=1$, what is the largest possible value of $|x|+|y|$?\n",
      "Let's think step by step\n",
      "If $(x,y)$ lies on the circle, \n",
      "so does $(x,-y),$ $(-x,-y),$ and $(-x,-y),$ (which all give the same value of $|x| + |y|$), \n",
      "so we can assume that $x \\ge 0$ and $y \\ge 0.$\n",
      "Then $|x| + |y| = x + y.$  Squaring, we get\n",
      "\\[(x + y)^2 = x^2 + 2xy + y^2 = 1 + 2xy.\\]\n",
      "Note that $(x - y)^2 \\ge 0.$  \n",
      "Expanding, we get $x^2 - 2xy + y^2 \\ge 0,$ so $2xy \\le x^2 + y^2 = 1.$  \n",
      "Hence,\\[1 + 2xy \\le 2,\\]which means $x + y \\le \\sqrt{2}.$  \n",
      "Equality occurs when $x = y = \\frac{1}{\\sqrt{2}},$ \n",
      "so the maximum value of $|x| + |y|$ is $\\boxed{\\sqrt{2}}.$\n",
      "The answer is \\sqrt{2}\n",
      "\n",
      "Question: If $f(x)=\\frac{ax+b}{cx+d}, abcd\\not=0$ and $f(f(x))=x$ for all $x$ in the domain of $f$, what is the value of $a+d$?\n",
      "Let's think step by step\n",
      "The condition $f(f(x))$ means that $f$ is the inverse of itself, \n",
      "so its graph is symmetrical about the line $y = x$. \n",
      "With a rational function of this form, we will have two asymptotes: \n",
      "a vertical one at $x=-d/c$ if $cx+d$ does not divide $ax+b$, \n",
      "and a horizontal one at $y=a/c$, \n",
      "if we take the limit of $f(x)$ as $x$ goes to $\\pm\\infty$. \n",
      "In order for $f$ to be its own inverse, the intersection of the asymptotes must lie on the line $y=x$ \n",
      "so that it and its asymptotes reflect onto themselves. \n",
      "This means that $-d/c=a/c$, \n",
      "and therefore $-d=a$ and $a+d=\\boxed{0}$.\n",
      "The answer is 0\n",
      "\n",
      "Question: A math teacher requires Noelle to do one homework assignment for each of the first five homework points she wants to earn; for each of the next five homework points, she needs to do two homework assignments; and so on, so that to earn the $n^{\\text{th}}$ homework point, she has to do $n\\div5$ (rounded up) homework assignments. For example, when she has 11 points, it will take $12\\div5=2.4\\rightarrow3$ homework assignments to earn her $12^{\\text{th}}$ point. What is the smallest number of homework assignments necessary to earn a total of 25 homework points?\n",
      "Let's think step by step\n",
      "Noelle only has to do 1 homework assignment to earn her first point, \n",
      "and the same is true for each of her first five points.  \n",
      "She must then do 2 homework assignments to earn her sixth point, seventh point, and so on, up to her tenth point. \n",
      "Continuing, we see that Noelle must do a total of \\[1+1+1+1+1+2+2+2+2+2+\\dots+5+5+5+5+5\\] homework assignments to earn 25 points.\n",
      "This sum may be rewritten as $5(1+2+3+4+5)=5(15)=\\boxed{75}$.\n",
      "The answer is 75\n",
      "\n",
      "Question: The quadratic equation $x^2+mx+n=0$ has roots that are twice those of $x^2+px+m=0,$ and none of $m,$ $n,$ and $p$ is zero. What is the value of $n/p?$\n",
      "Let's think step by step\n",
      "Let $r_1$ and $r_2$ be the roots of $x^2+px+m=0.$ \n",
      "Since the roots of $x^2+mx+n=0$ are $2r_1$ and $2r_2,$ we have the following relationships: \\[\n",
      "m=r_1 r_2,\\quad n=4r_1 r_2,\\quad p=-(r_1+r_2), \\quad\\text{and}\\quad\n",
      "m=-2(r_1+r_2).\n",
      "\\] So \\[\n",
      "n = 4m, \\quad p = \\frac{1}{2}m,\n",
      "\\quad\\text{and}\\quad\n",
      "\\frac{n}{p}=\\frac{4m}{\\frac{1}{2}m}=\\boxed{8}.\n",
      "\\]\n",
      "Alternatively, the roots of \\[\n",
      "\\left(\\frac{x}{2}\\right)^2 + p\\left(\\frac{x}{2}\\right) + m = 0\n",
      "\\] are twice those of $x^2 + px + m = 0.$ \n",
      "Since the first equation is equivalent to $x^2 + 2px + 4m = 0,$ \n",
      "we have \\[m = 2p \\quad\\text{and}\\quad n = 4m, \\quad\\text{so}\\quad \\frac{n}{p} = \\boxed{8}.\\]\n",
      "The answer is 8\n",
      "\n",
      "Question: Expand $(2z^2 + 5z - 6)(3z^3 - 2z + 1)$.\n",
      "Let's think step by step\n",
      "$$\\begin{array}{crrrrrrr}\n",
      "& & & 3z^3 & & -2z & + 1 & \\\\\n",
      "\\times & & & & 2z^2 & +5z & -6 \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& & & -18z^3 & & +12z & -6 & \\\\\n",
      "& & +15z^4 & & -10z^2 & +5z & & \\\\\n",
      "+ & 6z^5 & & -4z^3 & +2z^2 & & & \\\\\n",
      "\\cline{1-7}\\rule{0pt}{0.17in}\n",
      "& 6z^5 & +15z^4 & -22z^3 & - 8z^2 &+17z & -6 &\n",
      "\\end{array}$$ \n",
      "The answer is 6z^5+15z^4-22z^3-8z^2+17z-6}.\n",
      "\n",
      "Question: Find the mean of all solutions for $x$ when $x^3 + 3x^2 - 10x = 0$.\n",
      "Let's think step by step\n",
      "First, we factor the equation as $x(x^2 +3x - 10) = 0$.  \n",
      "So, one solution is $x=0$ and the other two solutions are the solutions to $x^2 + 3x-10=0$.\n",
      "We could either factor the quadratic, or note that the sum of the solutions to this quadratic is $-(3/1)=-3$, \n",
      "so the mean of the three solutions to the original equation is $-3/3=\\boxed{-1}$.\n",
      "The answer is -1\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(prompt_grade_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "8f13a11a-5d88-4292-9531-f9c412649105",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 [26:32<00:00,  7.96s/it]\n"
     ]
    }
   ],
   "source": [
    "i = 0\n",
    "with open('outputs/dev_algebra_grad_5_text_davinci_003.txt', 'w') as fd:\n",
    "    for d in tqdm(math_algebra_dev):\n",
    "        q = d['problem']\n",
    "        a = d['solution']\n",
    "        prompt_q = prompt_grade_5 + '\\nQuestion: ' + q + '\\n'\n",
    "        response = openai.Completion.create(model=\"text-davinci-003\", \n",
    "                                            prompt=prompt_q, \n",
    "                                            temperature=0, \n",
    "                                            max_tokens=256)\n",
    "        ans_model = response['choices'][0]['text']\n",
    "        fd.write('Q: %s\\nA_model:\\n%s\\nA:\\n%s\\n\\n' % (q, ans_model, a))\n",
    "        i += 1\n",
    "        # if(i == 10): break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2b26adc4-fb2a-4009-99d2-9faca4b94d40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_q 200 correct 41 ratio 0.2050\n"
     ]
    }
   ],
   "source": [
    "questions, ans_pred, ans_gold = parse_pred_ans(\"outputs/dev_algebra_grad_5_text_davinci_003.txt\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
